E-RNN: Entangled Recurrent Neural Networks for Causal Prediction

We propose a novel architecture of recurrent neural networks (RNNs) for causal prediction which we call Entangled RNN (E-RNN). To issue causal predictions, E-RNN can propagate the backward hidden states of Bi-RNN through an additional forward hidden layer. Unlike a 2-layer RNNs, all the hidden states of E-RNN depend on all the inputs seen so far. Furthermore, unlike a Bi-RNN, for causal prediction, E-RNN depends on both the forward and backward hidden states. Importantly, E-RNN is a general architecture that can be combined with various RNN techniques such as multi-layer, dropout, and GRU. Using three real-world datasets, we show that E-RNN significantly and consistently improves the performance of previous RNN architectures with the same complexity.